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Abstract 

The quantitative methodologies have been traditionally employed in the educational research so 
far. However, as long as with the appreciation and widespread use of the qualitative methodologies 
in many disciplines, many different educational areas have started to be examined in terms of 
qualitative research aspects. Particularly, the qualitative evaluation of the education programs has 
received considerable interest and there have been recently some attempts to develop a qualitative 
methodology for evaluating educational programs based upon the tenets of program evaluation. 
The evaluators have underlined the benefits of qualitative methods to boost the information shared 
with decision-makers and policy makers. The most inclusive endeavour has been carried out by 
Eisner. Eisner's program evaluation model presents the role of educational connoisseurship and 
criticism in educational evaluation in terms of qualitative evaluation. This study aims at examining 
how a qualitative program evaluation is conducted in relation with the Eisner's evaluation model. 
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Introduction 

The program evaluation can be briefly defined as a process of contribution to the development of 
education program, decision on a program, and describing the current situation through the 
evaluation of application process (Saglam & Yuksel, 2007). Throughout the historical development of 
the evaluation, the quantitative methods were dominantly used for the evaluation of education 
programs while there were very few qualitative studies. The benefits of the quantitative methods used 
in the evaluation cannot be denied and in literature it is emphasized that with well-designed 
quantitative studies, there are many advantages of the quantitative methods, such as the opportunity 
to decide on larger populations using the samplings, the possibility to reanalyze the research findings 
by other researchers (meta analysis) (Sandelowski, Docherty, & Emden, 1997). Furthermore, many 
researchers depict the advantages of quantitative methods by describing the weaknesses of 
qualitative methods. Quantitative methods grasp the guarantee of correcting the weaknesses of 
qualitative methods. Unlike traditional qualitative approaches, which tend to be anecdotal, 
noncomparative, atheoretical, too legalistic, too descriptive, quantitative methods are generalizable, 
comparable, theory-based and explanatory (Bogdan & Biklen, 1998; Eisner, 1991; Howe, 1992). 

On the other hand, these strengths of the quantitative studies can be disadvantages in social science 
research and can cause some weaknesses in the studies, for instance; the issues such as perceptions, 
beliefs cannot be explained with numbers effectively or cannot be understood sufficiently since the 
context is not be described exactly (Ambert, Adler, Adler, & Detzler, 1995). However, qualitative 
approaches highlight the importance of looking at issues in their usual setting and collecting 
comprehensive data through techniques such as case studies, participant observation, descriptive 
narratives, focus groups, and in-depth interviews (Ting-Toomey, 1984). The researchers search for an 
in depth, overall description of the subject and context in a qualitative study (Denzin & Lincoln, 1998; 
Greene, 1994). Yet, many evaluation studies are unfortunately conducted at different platforms 
instead of the actual context they should be, the evaluator stands away from the program, thus this 
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case damages the quality of the evaluation and decreases its reliability. However, it is quite crucial to 
examine the program process in all parts in order to judge about a program. On the basis of the 
definitions of the qualitative methods, it can be stated that qualitative methods could address this gap 
in the program evaluation. 

The use of qualitative method in evaluation studies was based on the emergence alternative program 
evaluation models in 1960s and 70s (Mathison, 2005). In this period, the qualitative method was 
either alone or together with the quantitative method applied in the studies. In the following years, 
Cronbach (1982), who is one of the pioneers of the quantitative method, stated that the qualitative 
evaluation is important and emphasized the significance of observations and interviews to determine 
the success of the program. Thus, he underlined that as well as the quantitative methods, the 
qualitative research methods should be used in the research. However, it should be remarked that for 
every program evaluation, the qualitative methods cannot be appropriate. Therefore, it is important 
to define that the program would be evaluated on the basis of which program evaluation method. For 
instance, for an evaluation study in which expertise-oriented program evaluation approach is based, 
applying the quantitative methods can be very difficult (Hoepfl, 1997). In this context, in such a study, 
it would be more appropriate to examine the qualitative data collection methods and to probe one of 
the qualitative program evaluation models; Educational Connoisseurship and Criticism model 
developed by Eisner. 


Qualitative Program Evaluation Method 

Qualitative research method uses a naturalistic approach to understand phenomena in context-specific 
settings, such as real world setting [where] the researcher does not attempt to manipulate the 
phenomenon of interest (Patton, 2002). In other words, qualitative research can be broadly defined as 
"any kind of research that produces findings not arrived at by means of statistical procedures or other 
means of quantification" (Strauss & Corbin, 1990). Qualitative evaluations use qualitative and 
naturalistic methods, sometimes alone, but often in combination with quantitative data. The data for 
qualitative evaluation typically comes from fieldwork. The evaluator spends time in the setting under 
study— a program, organization, or community where change efforts can be observed, people 
interviewed, and documents analyzed. Qualitative methods include three kinds of data collection: in- 
depth, open-ended interviews; direct observation; and written documents (Denzin & Lincoln, 1998; 
Greene, 1994). 

Observations: Qualitative observations depend more on evaluator or observer. In the process, 
checklists may be used, but typically, they are less structured (Fitzpatrick, Sanders, & Worthen, 2004). 
The evaluator makes direct observations of activities and interactions, sometimes engaging personally 
in those activities as a "participant observer." For example, an evaluator might participate in all or part 
of the program under study, participating as a regular program member, client, or student. Fieldwork 
descriptions of activities, behaviours, actions, conversations, interpersonal interactions, organizational 
or community processes, or any other aspect of observable human experience is included in 
observations. The stages of qualitative observations include; 

• preparation through reading, chatting with informants 

• articulating the purpose of observation 

• looking at what occurs in program 

• listening the participants 

• synthesizing the information 

• checking with hypothesis 

• confirming and cross-checking 

Documents: the evaluator's first consideration for sources and data collection methods should be 
existing information such as documents or reports. Existing information is cost-effective and not 
corrupted by the participants (Fitzpatrick et al., 2004). Written materials and other documents from 
organizational, clinical, or program records; official publications and reports; personal diaries, letters, 
artistic works, photographs, and memorabilia; and written responses to open-ended surveys are 
samples for documents. 
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Interviews: Interviews are often a key to qualitative data collection. Although observations are 
typically the core elements of qualitative evaluation, there is so much that evaluator cannot observe 
(Fitzpatrick et al., 2004). The qualitative evaluator talks with people about their experiences and 
perceptions. Thus, the qualitative data for evaluation is collected with the observations, interviews, 
and document reviews and analyzed with major themes, categories, and case examples through 
content analysis, which requires considerably more than just reading to see what is there. These 
themes, patterns, understandings are the products of qualitative evaluation. Findings of the evaluation 
may be presented alone or with quantitative data. Generating useful and credible qualitative findings 
through observation, interviewing, and content analysis requires discipline, knowledge, training, 
practice, creativity, and hard work (Greene, Caracelli, & Graham, 1989). Considering this, the basic 
reason why the qualitative methods are used in program evaluation studies can be explained as to 
reveal the entire process and to reveal the whole story of program with information collected from the 
participants. Only qualitative studies provide in-depth information about what happens at the program 
application and evaluation process, with whom and what kind of communication is established, and 
under what conditions the program is applied. Through the findings of such studies, it could be 
possible to examine the story of the program and the experience of the participation during the 
evaluation process. Moreover, such studies could inform the individuals, who would decide on the 
future of the program, about the evaluation process and results (Davies & Dodd, 2002). 

Educational Connoisseurship and Criticism Model 

Educational Connoisseurship and Criticism Model was developed by Eisner, on the basis of expertise- 
oriented program evaluation approach, which grounds on the professional expertise of the program 
evaluators while evaluating an institution, program, product or activity (Eisner, 1976). This approach 
can be used in broad context from education to different areas in accordance with the evaluand and 
the expertise of the evaluator. In this sense, Eisner (1985) explained that this model addresses to a 
form of educational inquiry which is qualitative in character and sources from the work of the critics 
done in literature, theatre, visual arts and education. 

Considering the examples of evaluation in real life, it can be stated that for the evaluation of an 
education program, a program development and evaluation experts are necessary while for the 
evaluation of the effectiveness of a hospital services, doctors, staffs, pharmacist etc are required. 
Moreover, the steering committee for PhD degree, accreditation units evaluating institutions and firms, 
referee committee for article reviews, commissions evaluating the program on-site are some examples 
for this evaluation approach. The examples indicate that this approach is applied through formal and 
informal investigations, panels and individual examinations, which can be accepted as the tools for 
gathering qualitative data. 

Specifically, for the context of program evaluation, this approach is represented by Educational 
Connoisseurship and Criticism Model. "Educational Connoisseurship" and "Educational Criticism" are 
the two basic concepts of the model. In fact, both of these concepts are related to art. According to 
Eisner (1976, 1985), the aim of the expertise, which is defined as "the art of appreciation and 
evaluation", is to reveal the awareness of the qualifications composing a process or an object and to 
emphasize them. For example, at theatre, the expertise can be focused on the stage, plot, 
performances of players, and pace but in education, the quality of program, students' activities, 
quality of education, learning processes and equipment so forth can be focused. On the other hand, 
criticism, this was defined by Eisner (1976, 1985) as "the art of disclosing the quality of events or 
objects that connoisseurship perceives". In order to share the connoisseurship, criticism is required. 
Contrary to common meaning of criticism, rather than making negative comments, criticism refers to 
reproduce the perception of the object. Like an art critic, who attempts to provide different viewpoints 
on sculpture or painting and to make them comprehensible, an education critic wants to reveal the 
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events in the class such as class rules, the quality of education, changes in students' behaviours. 
According to Eisner (1976), an expert has not got a role of critic but he evaluates the works and 
appreciates. On the other hand, a critic requires an expert to do the required works. Briefly, the 
awareness related to the program, project, and effect of teaching is duty of expertise while to 
announce this awareness to public is the duty of critic. Eisner used these two concepts within the 
context of education and proposed Educational Connoisseurship and Criticism Model. 

By emphasizing that teaching requires artistic skills, Eisner stated that education is a cultural art and 
this is a process differing from an individual to another or from one environment to another (1985). In 
this context, he defined the aim of the educational evaluation as not only to review the products or 
evaluate the activities within the process but also to increase the skill that a teacher would gain. 

Eisner added the concepts of connoisseurship and criticism to the evaluation literature as a result of 
general trend in the USA and Europe in 1970s. In those years, it was realized that quasi-experimental 
methods and other scientific methods are not so appropriate for educational evaluation. Due to the 
growing doubts about inadequacy or inappropraicy of scientific evaluation approaches, many 
education experts reconsidered the style and function of the evaluation. These experts got influenced 
from different academic disciplines such as anthropology, journalism, philosophy and law etc and 
hence they developed many evaluation models like naturalistic-constructivist evaluation, the 
evaluation responding to the needs, goal-free evaluation and adversary models (Mathison, 2005). 
Eisner got influenced from this trend and proposed a model on the basis of art and aesthetic. 

In Eisner's (1976) model, the program evaluator resembles to the art expert and the evaluation 
process to the art criticism. In this context, while an evaluator is doing educational criticism on a 
program, class or school, firstly he describes what he sees, then interprets and lastly evaluates 
(Eisner, 1976, 1985). This model is developed on three main dimensions reflecting three qualitative 
actions. These dimensions are: 

Descriptive Dimension: According to Eisner (1976), the descriptive dimension of educational 
criticism is related to describing the current state of program, class and school etc. Eisner 
(1985) explained that "verbal statement should be sharp" in the descriptive dimension. 
Therefore, like in art criticism, language and figure speeches are used as emphasis in 
educational criticism while describing the aesthetic dimensions of evaluand. The descriptive 
educational criticism informs about the number and type of the questions in class, the 
discussion time and the impression of the class on the student. 

Interpretative Dimension: Eisner (1985) stated that the interpretative dimension of the 
educational criticism is related to the attempt to understand the meaning and significance of 
many activities in social environment. This dimension reveals the expert's knowledge of using 
multiple theories, viewpoint and models while interpreting the activities at education 
environments (Koetting, 1988). For instance, a critic should answer the interpretive questions 
such as how the teacher and students interprets the raising hands in class, what means class 
environment for all participants etc. Eisner specified that such activities should be commented 
by the evaluator and for this; there should be large theoretical knowledge and series of 
criterion. 

Evaluative Dimension: The last dimension of educational criticism is the evaluation. In this 
dimension, the educational significance and effect of the interpreted experience/activities are 
evaluated. During this process, there should be some educational criteria to judge about the 
experience. According to Koetting (1988), this situation addresses to the normative feature of 
the educational criticism. 
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Combining the expertise/connoisseurship and criticism, Eisner's perspective on qualitative inquiry 
within the framework of Educational Connoisseurship and Criticism model addresses and allows for a 
diversity of teaching methodologies, alternative measurement and evaluation methods, participation 
of students in the overall education process. Briefly, this qualitative model conceptualizes the whole 
picture of education through qualitative analysis and evaluation. 

Conclusion 

This paper introduced why qualitative methods are important, which qualitative data collection 
techniques are applicable in program evaluation process, and made clear the qualitative features of 
Eisner's program evaluation model. The study does not examine which method is more effective 
evaluation, but scrutinizes the importance of employing both approaches in collaboration and 
highlights the worth of "words" in program evaluation studies. Thus, quantitative data are "the 
numbers" collected through surveys or other measurement techniques. Qualitative data are "the 
words" collected through interviews; focus groups, participant observation, or related methods. 

"The words" lie at the underneath of Eisner's program evaluation model. Since the evaluations carried 
in accordance with this model typically based on evaluators' criticisms and so on the words, the model 
strongly has qualitative evaluative aspects. In the evaluative aspect of model, the critic or evaluator 
makes an assessment of the educational importance or significance of the experience he/she has 
described or interpreted verbally. However, the educational connoisseur/educational critic understand 
the value of his judgments. As is the case with any criticism, disagreement with any aspect 
(descriptive, interpretive, and evaluative) of the criticism is open to debate. Eisner sees this as 
strength of qualitative evaluation. It can be inferred from the Eisner's studies that there are two forms 
of qualitative evaluation in education. Evaluators as critics use qualitative forms when they become 
involved in making statements about reality as in arts. The result is a qualitative program. The second 
form is found in the studies of evaluators. The evaluator finds himself with the difficult task of 
rendering the indefinable qualities constituting educational program into words that will help intended 
users appreciate value, understand and apprehend the program more intensely. 

To sum up, through the introduction and application of Eisner's Educational Connoisseurship and 
Criticism Model, the program evaluation can gain a qualitative stance and thus the product and 
process of the program can be evaluated and criticised from larger, diverse perspectives, independent 
from the number limitations. 
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